Not Applicable
Not Applicable
Not Applicable
This invention relates to image data storage, especially to an improved method of addressing high resolution image data that is stored in a plurality of frame buffers where each of the frame buffers are coupled to one or more video streams.
It is well known that the majority of computers and software applications in current operation utilize a graphical user interface (GUI) due to the relative ease of use provided compared to older text based user interfaces. Certain users of computer systems and display systems benefit from the use of multiple display devices to allow more image data to be displayed simultaneously. For example, this can be for a single software application to have a display area that spans multiple display devices to effectively create a larger higher resolution display. Alternately, a user may desire to have multiple software applications or display windows distributed across the multiple displays, or a computer can be used for multiple users with each of the display devices displaying the software applications of the respective users.
In order to allow display controllers of varying types to be utilized in a computer system, computer operating systems utilize software often referred to as a device driver that is specifically written for a type of display controller. A device driver for a display accepts standardized hardware independent graphics commands from the operating system or from a software application, the device driver then generates the appropriate hardware dependent commands or memory accesses to load the image data into the video memory. Certain older computer operating systems such as Microsoft Windows 3.1 required the device driver to implement all the software functionality required by the graphics commands. Newer operating systems recommend for a device driver to provide only the software functionality for the graphics commands that can be accelerated by the display controller allowing for less device driver software to be written increasing reliability and decreasing development time. An operating system software component such as the DIBENGINE of Microsoft Windows 95 processes the graphics commands not handled by the device driver. Such a device driver is sometimes referred to as a mini-driver. In order for the operating system to be able to process graphics commands, the operating system specifies requirements for the display controller and the memory used for storing display surfaces, such a requirement is for the surface memory to be addressable as a frame buffer. By providing a surface memory that is accessible as a frame buffer the benefits of implementing a mini-driver can be obtained, this is without regard to whether a single image is displayed across the surface or if multiple display windows are distributed across the surface. Various operating systems including, but not limited to Windows 95, Windows 98, Windows CE 2.0 and Windows NT 4.0 recommend the use of a display device that provides a frame buffer. One reference describing the benefits is Microsoft Windows 95 DDK (Device Driver Kit) under the section titled Display Mini-driver Guidelines.
A frame buffer is often defined as a block of addressable memory space for storing image data, where the first address of a collection of addressable locations for storing a pixel is determined from the X coordinate, the Y coordinate, the pixel size or number of addressable locations for storing a pixel, a base address, a pitch and a start address offset from the base address to the storage of the pixel having X and Y coordinates of zero. The pitch is the difference in addresses between the storage of adjacent scan lines which can be greater than the width of a scan line where the width is the number of addresses for storing the pixels in a scan line. The collection of addressable locations for storing a pixel can contain one or more addressable locations, for example a video mode that uses a palette can use a single byte that stores an index into a color table, alternately the collection can contain three bytes for storing the red green and blue components of a color. A frame buffer by this definition differs from a video memory in that a video memory can contain multiple frame buffers for example to store multiple display surfaces.
A display surface that can include the entire graphics area accessed by the operating system for a particular video mode can be used as the primary display surface by the operating system, where the video mode specifies the horizontal resolution, vertical resolution and the pixel storage format. This graphics area is sometimes referred to as the desktop of the GUI. The desktop can include a larger area than that viewable at one time by a display system, this is sometimes referred to as a logical desktop with the user given access to software controls for panning to a desired area within the logical desktop.
Another benefit of a display system providing a frame buffer is that software applications can bypass the Graphics Device Interface (GDI) and device driver and access the primary display surface memory directly maximizing the speed of updating the image data. Display surfaces created for use in Microsoft""s DirectDraw are accessible as frame buffers. A reference describing how an application can access a display surface is in Microsoft""s Platform SDK and is titled Accessing Surface Memory Directly and is under Graphics and Multimedia Services in the Microsoft DirectX 6.0 section.
On processors such as the Intel 386, 486 and Pentium types or compatibles, logical addresses are added to a segment base address to form what are referred to as linear addresses. The linear addresses are then mapped to physical addresses via page directories and page tables. Computer systems containing memory management hardware with address translation capability such as this allow a virtual memory system to be provided that allows more memory to be seen by the software than is contained in the computers RAM by switching or paging between memory blocks contained on disk and in RAM. Switching is based on whether linear pages are marked as present or not present in RAM. When an address is generated by software that is not present in RAM a page fault is generated and a page fault exception handler performs a page switch and the faulting memory access is re-executed all transparently to the application software. This however results in a reduced execution time due to the page fault handling software and the relatively slow disk access time compared to RAM access time.
For display controllers with a single physical frame buffer, a contiguous number of linear address pages are sequentially mapped to contiguous physical pages in the frame buffer. Most current technology display controllers for driving a single display device provide access to video memory as a frame buffer. However, certain older technology display controllers for driving a single display device do not provide a frame buffer for accessing the video memory, but provide access to the video memory by a technique known as bank switching. Bank switching is used to allow a CPU to access one megabyte of video memory for example by alternately switching one of sixteen banks of sixty-four kilobytes each into the memory address region A0000 to AFFFF to allow software running in real mode to access the entire video memory. When Microsoft introduced Windows 95 there were still many personal computers (PCs) in use that utilized these older technology banked switched display controllers, so in order to allow these bank switching display controllers to utilize a mini-driver, Microsoft provides software known as a Virtual Flat Frame Buffer Device (VFLATD). The VFLATD copies bank switching software written for a specific display controller into a page fault handling routine to allow the page fault handler to switch to the appropriate sixty-four kilobyte bank as needed. This however results in reduced graphics performance due to the necessity of processing the bank switching routines. This performance degradation is particularly severe when pixels along a column of a display are sequentially accessed. A reference on VFLATD is Microsoft""s Windows 95 DDK.
A graphics aperture remapping table (GART) is a known address translation method used in Accelerated Graphics Port (AGP) devices for accessing the computer system""s RAM memory as graphics memory in the xe2x80x9cExecutexe2x80x9d usage model. This RAM memory is typically used for storing 3D textures. When the CPU allocates a block of system memory that will be used as AGP graphics memory, there is no guarantee that the memory will be allocated in contiguous pages of RAM memory. The CPU uses its page directories and page tables for accessing the block of memory as if the block was in contiguous memory space. The AGP devices use bus mastering for accessing the system RAM, but the CPU""s page directories and page tables are not available to the AGP device for making the memory block contiguous. The GART is an address translation method in an AGP device that is used to make the memory block contiguous to the AGP bus master. The GART output addresses that result from the translation are dictated by the CPU""s page directory and page table setup for accessing the block of memory. The GART can be configured such that input addresses to the GART utilized by the AGP device make the block of system RAM map to the end of the local video memory of the AGP device making the devices video memory appear larger. The configuration of the GART just duplicates the page ordering performed by the CPU page directories and page tables, so that the block of memory allocated by the CPU also appears contiguous to the AGP device. A reference on the GART is the Accelerated Graphics Port Interface Specification Revision 1.0 in the section Two Usage Models: xe2x80x9cExecutexe2x80x9d and xe2x80x9cDMAxe2x80x9d.
A dual display device board called the Duet and manufactured by Appian Graphics is known that utilizes the Virge/MX display controller for generating two video outputs using a single frame buffer and a single display controller. Though this board is capable of utilizing a mini-driver for the two displays, the cost of the board is over four times the cost of two comparable single display controller boards. Besides cost, another disadvantage of such an approach is that video memory must be accessed at very high rates to drive the two displays. This prevents the use of the higher resolutions. Also the number of display devices is limited to two for the single frame buffer.
Multiple display controller boards for driving multiple display utilizing custom designed PCI-PCI bridge circuits to access the multiple display controllers and their associated video memories have a disadvantage of high development costs that result in high prices for these multiple display controller boards.
Another known multiple display approach sold by Stone Microsystems is the AEON TwinS model. This utilizes multiple standard display controller boards each with a video memory of two megabytes. It is apparent that this approach does not utilize a single high resolution frame buffer for accessing the multiple frame buffers as the display modes and multiple display arrangements specified as being supported could not be implemented with address translation based on a page size of 4096 provided by the Intel processors when used with a video memory size of two megabytes per display controller.
Microsoft Windows 98 is the first version of Windows to provide built in support for multiple display devices. A software component called the Display Driver Management Layer (DDML) has been added to the GDI and is used for large desktop support allowing images to span multiple displays. DDML separates the GDI commands into commands for multiple display device drivers and multiple frame buffers. As the DDML uses a display driver and a frame buffer for each of the multiple graphics adapters it turns off device bitmaps, as there is no single frame buffer to write the bitmaps to. Demanding graphics applications for Windows 98, such as computer games and multimedia, often utilize Microsoft""s DirectDraw for drawing to a display device as it provides higher performance graphics than is available using the Graphics Device Interface (GDI). Two types of DirectDraw graphics applications are possible, Full-Screen exclusive mode and Normal mode. The Full-Screen type of DirectDraw application gives the application fall access to the display surface, in other words there is no windows title bar or window control buttons as in normal windows applications. Also DirectDraw Full-Screen applications can perform page flipping with the primary surface used for the desktop and can change the palette of the primary surface. The disadvantage of Windows 98 with DirectDraw for Full-Screen exclusive mode applications is that multiple frame buffer spanning is not supported or in other words applications cannot be Full-Multi-Screen applications preventing the use of page flipping across the multiple display desktop. Without page flipping the quality of displaying motion video and animation across the multiple frame buffers is greatly reduced. The Normal type of DirectDraw application appears like a normal windows application with a title bar and window control buttons. With the DDML, when Normal type DirectDraw applications span multiple display devices DirectDraw does not use hardware acceleration, but uses software emulation that decreases graphics performance. One other disadvantage of DDML is that it is not available for use under Windows 95. A reference on DDML is the presentation titled Multiple Monitor Support [HV Track from Meltdown 97. A reference on DirectDraw acceleration in multiple monitor systems can be found in Microsoft""s Platform SDK and is titled Devices and Acceleration in MultiMon Systems and is under Graphics and Multimedia Services in the Microsoft DirectX 6.0 section.
Image data stored in video memory surfaces are typically generated by the CPU, however image data may also be obtained from video sources such as, but not limited to cameras, MPEG/DVD decoders and digital television signals. This image data can be displayed if desired as video overlays without overwriting the image data corresponding to the PCs desktop and optionally, still images or motion video can be captured without being displayed. As the resolution of a video source device is increased, the cost of the device also increases as well as the cost of an image acquisition system to interface to the device. By using an array of image sensors with a single lens system or an array of cameras and lenses with a calibrated optical system, a high resolution image can be obtained by using multiple video signals to represent the high resolution video and by using common video sync signals. A system to display and acquire image data from the multiple video signals can be economically provided with multiple displays and multiple display controllers each having a video digitizer with the image data distributed among the video memories associated with the multiple display controllers. Though economical with respect to hardware costs, a disadvantage of such an approach is that existing operating system software, such as Microsoft""s DirectDraw video port extensions that require the video data to reside in a single frame buffer, cannot be used. Therefore custom application software must be written to capture the high resolution still images or sequences of high resolution video frames.
It is therefore an object of the present invention to provide a method and apparatus for addressing multiple frame buffers as a single high resolution logical frame buffer. Therefore decreasing the complexity of software for interfacing to the multiple frame buffers, thereby decreasing development time and costs and increasing software reliability. It is also an object of the invention to provide a low cost multiple display system by utilizing standard commercially available components. Creating a single frame buffer allows the use of existing operating system software components to operate on the high resolution image data minimizing the amount of custom software that must be written for accessing the multiple frame buffers. Storing the high resolution image data in multiple frame buffers enables the high resolution image data to be displayed utilizing multiple standard resolution video outputs or enables the acquisition of the high resolution image data from multiple standard resolution video sources. For the purposes of this invention, the term high resolution is relative to the lower resolution video streams being combined to form a logically higher resolution device. An embodiment of the invention distributes image data corresponding to a standard display resolution across multiple standard display devices for the purpose of creating a display that is larger in size. For this embodiment, though the image data is not high resolution, the display resolution is still increased by utilizing interpolated zooming before displaying the image data.
The fact that the image data is stored in multiple frame buffers in physical address space prevents it from being a single high resolution frame buffer in physical address space. This is because the pixel storage for adjacent pixels within a logical scan line of the high resolution image will exhibit a physical address discontinuity when crossing between frame buffers in a two column arrangement of frame buffers. However, this is also for a single column arrangement of frame buffers where logical scan lines contain only pixel data from a single physically addressable frame buffer. Here there is a discontinuity between the storage of the last pixel in the last scan line of a frame buffer and the storage of the first pixel in the first scan line of the frame buffer below it. This is due to display controllers having more video memory than required for a particular video mode. Also contributing to this address discontinuity is display controllers typically provide address space for memory mapped input output (NMO) as well as often providing address spaces for both Intel type (little-endian) byte ordering and Motorola type (big-endian) byte ordering.
Despite physical address discontinuities at frame buffer crossover boundaries, addressing of the multiple frame buffers as a single logical frame buffer is provided for by the current invention. This is provided by first dividing the image data into pages using a page size appropriate for both the video mode and arrangement of the physical frame buffers within the high resolution image. Then a pitch is determined for each of the physical frame buffers that enables the alignment of the memory pages at the frame buffer crossovers. Then for video modes utilizing multiple bytes per pixel, the collection of bytes representing the pixels are aligned on the page boundaries at the frame buffer crossovers. Then linear address space is reserved for storing a single high resolution frame buffer. Then address translation hardware is configured to shuffle the mapping of the pages such that the pages within the reserved linear address space are routed to the appropriate pages within the multiple physical frame buffers to create a single high resolution frame buffer when accessed with an appropriate logical pitch.
By creating a single high resolution logical frame buffer accessible by CPU logical addresses, one advantage provided is the task of writing a device driver for accessing the multiple frame buffers is significantly reduced in complexity. The single logical frame buffer enables the use of what is known as a mini-driver which allows an operating system software component such as, but not limited to, the DIBENGINE of Microsoft Windows 95/98 to access the multiple frame buffers in a device independent manner. The operating system component by way of the single logical frame buffer handles any graphics commands that the device driver chooses not to handle. For example, to maximize performance the device driver only handles the graphics commands that benefit from hardware acceleration by the device driver implementing the necessary hardware dependent instructions for these commands. This solves the problem of having to write device driver software to process every graphics command which would increase development time and costs, decrease software reliability and reduce graphics performance by having to implement extra software for dividing the graphics commands into commands for each of the frame buffers.
Another advantage is the single logical frame buffer can be accessed directly by software applications such a DirectDraw application. This prevents having to perform graphics commands to access individual pixel data that is much slower then accessing the frame buffer directly.
Another advantage of creating a single logical frame buffer is for the case of interfacing to multiple video sources to effectively create a higher resolution video source. By providing a single logical frame buffer an existing standardized low level software interface of an operating system intended for single video sources can utilize the single logical frame buffer as the target surface for a logical high resolution video port. The low level software interface can be accessed by software applications for controlling the display and capture of data from the logical high resolution video port. However, more preferably the low level software interface is accessed by a standardized higher level software interface allowing control of the logical video port from software applications with simple to use high level instructions. An example of a low level software interface is Microsoft""s DirectDraw video port extensions and an example of a high level software interface is that provided by Microsoft""s DirectShow. This solves the problem of having to write custom low level and high level software for interfacing to the multiple frame buffers as a single high resolution video port.
Though the use of custom PCI-PCI bridge circuitry results in high costs for multiple display controller boards, it does provide the benefit of allowing more display controllers to be installed in a given number of computer expansion slots. By incorporating page shuffling address translation circuitry into a custom PCI-PCI bridge and by selecting a page size that allows an integer number of pages in the width of a desired video mode, the current invention provides the advantage of making optimum use of video memory.
Though Microsoft Windows 98 has support for multiple displays without requiring a single logical frame buffer by using the DDML, providing a single logical frame buffer provides advantages when DirectDraw is used with Windows 98. For the type of DirectDraw applications known as full-screen exclusive mode, Windows 98 limits these applications to a single frame buffer. In other words, these applications cannot be full-multi-screen applications that span the multiple frame buffers. The use of the single logical frame buffer provides the advantage of allowing full-screen exclusive mode applications to be full-multi-screen applications that span the multiple frame buffers enabling the use of page flipping across the multiple frame buffers which is essential for high quality video playback and animation. Providing the single frame buffer also provides the advantage of enabling the other features of this DirectDraw mode to be used across the multiple frame buffers, such as allowing the application to change the palette of the primary surface or change the effective resolution of the multiple displays. Another type of DirectDraw application uses what is known as the normal cooperative level. This type of DirectDraw application looks like a normal Windows application with a movable and resizable display window. A disadvantage of Windows 98 and the DDML with respect to this type of DirectDraw application is that hardware acceleration is not used when the display window spans two or more frame buffers. Microsoft recommends to maximize performance of this type of DirectDraw application that the application keep track of which part of their display window resides on which frame buffer and perform separate BitBLTs (bit block transfers) to each device. A disadvantage of this is the software applications need to be written specifically for the multiple frame buffer system. This prevents the performance from being maximized for existing software applications. An advantage of using a single logical frame buffer is the applications do not to have to perform this processing to maximize performance. The performance gain is accomplished by incorporating the software to perform multiple frame buffer BitBLTs into a DirectDraw HAL (Hardware Abstraction Layer) as well as incorporating any other DirectDraw functions for which hardware acceleration is available.